This invention relates to a method and an apparatus for image processing for noise suppression and sharpness enhancement of digital images, or specifically to a method and an apparatus by which noise such as graininess in digital images can be suppressed while enhancing their sharpness.
In the present invention, the noise in digital image is called xe2x80x9cgrain or graininessxe2x80x9d in view of the appropriateness of the expression if the image reproduced in photographs, printed documents and so forth is discussed and the noise region of a digital image being processed which consists of a noise component and which is to be separated from the edge region which consists of an edge (contour) component is called xe2x80x9ca grainy or flat regionxe2x80x9d in view of the appropriateness of the expression if the image reproduced in photographs, printed documents and so forth is discussed.
In digital imaging technology which obtains pictures such as photographs with an image input scanner and which outputs digital images with an image output printer, considerable deterioration occurs in the sharpness of the output image due to the scanner and the printer. As a corrective procedure, sharpness enhancement is conventionally performed by means of a Laplacian filter or an unsharp masking (USM). However, sharpening the image causes the side effect of increasing noise such as graininess and, hence, grainy pictures can be subjected to only moderate sharpness enhancement within a range where graininess deterioration is tolerated; as a result, it has been difficult to obtain image quality better than that of the original grainy picture.
Several methods have been proposed to process digital images such as to remove noisy graininess and enhance their sharpness. Removal of graininess involves an averaging or blurring technique but the blurred grainy pattern is not pleasing to the eye or fine structures of the object may be erased in an unnatural way. For these and other reasons, the conventional techniques for removing graininess are not suitable for application to high-quality pictures such as photographs.
Pictures such as those in photographs, printed documents, or on television""s screens and from various kinds of copiers suffer diverse deterioration problems, i.e., sharpness deterioration due to optics such as a camera, graininess and sharpness deterioration inherent in photographic materials, or noise and sharpness deterioration that is added when the original picture such as a photograph or a printed document is digitized with an image input device. In order to deal with these difficulties, various methods have been proposed to process images such as to reduce noise and enhance their sharpness. Smoothing and coring are two common methods employed in the conventional image processing technology for removing graininess, whereas sharpness enhancement is implemented by unsharp masking (USM) or processing with a Laplacian or a high-pass filter. However, if graininess is suppressed by these conventional methods of graininess removal, artifacts that cause unnatural and strange impressions will occur or fine structures of the image that should inherently be kept intact will be suppressed along with the graininess. See, for example, Japanese Domestic Announcement (koho) Nos. Sho 57-500311 and 57-500354, as well as P. G. Powell and B. E. Bayer, xe2x80x9cA Method for the Digital Enhancement of Unsharp, Grainy Photographic Imagesxe2x80x9d in the Proceedings of the International Conference on Electronic Image Processing, Jul. 26-28, 1982, pp. 179-183. According to the method proposed by Powell and Bayer in these references, suppression of graininess is accomplished by smoothing (with a low-pass filter) and sharpness enhancement is performed with an unsharp masking (high-pass filter). In the smoothing process, signal values for nxc3x97n pixels are multiplied by Gaussian or other type of weights such that the signals are smoothed to suppress graininess. In the sharpness enhancement process, picture signals for mxc3x97m pixels are first used to determine differential values by calculation from the central pixel towards the surrounding pixels and if any differential value is smaller than a preset threshold, the pixel of interest is regarded as representing graininess or noise and removed by coring and the remaining differential values which are greater than the threshold are summed up, multiplied by a constant more than 1.0 and added to the previously smoothed signals, thereby achieving sharpness enhancement.
In this process, the density contrast of grainy patterns decreases since they are blurred; on the other hand, blurred grainy patterns may become visually pronounced as they are mottles of randomly crowded grains that cause graininess (this phenomenon is commonly referred to as xe2x80x9cmottlingxe2x80x9d) and they will present unpleasing graininess. In addition, a preset threshold is used as a criterion for distinguishing graininess from the picture (this is the coring process), so image signals of low contrast may occasionally be erroneously taken as graininess and suppressed or removed along with the latter or discontinuity may occur at the boundary between the removed image signal and the enhanced picture signal to produce an unnatural artifact in the output image. This drawback occurs very frequently in fine images such as those of lawn and carpets and in images that represent texture as in fabrics and the result is an artifact that is visually quite unnatural and hence undesirable.
In the above-described prior art method of processing images to suppress their graininess while enhancing their sharpness, unsharp masking is employed to enhance the sharpness whereas blurring or smoothing is effected to suppress the graininess, such that a graininess (noise) signal and a contour signal are separated from the original picture by signal level and the contour signal is subjected to sharpness enhancement whereas the smoothed region is suppressed in graininess and the smaller signal is regarded as representing graininess and processed accordingly; as a result, signals representing the fine details of the picture which are close to signal levels representing graininess, namely, image signals representing the texture of cloths, the hair on the head and the like, will be suppressed along with the graininess, yielding visually unpleasing images that contain artifacts from image processing In the conventional image processing technology where blurring or averaging is used as the method of suppressing graininess, a blurred grainy pattern is reduced in terms of density fluctuation; on the other hand, blurred grainy pattern spreads despite the small amount of density fluctuation and will be recognized as a visually unpleasing pattern, which stands out markedly in someone""s face or skin as in portraits or in solid objects such as walls or sky.
In the prior art, a grainy (noisy) signal region and a contour region are separated from the original picture by signal level. Stated more specifically, the contour region and a flat region are discriminated on the basis of a signal indicating the difference between the original picture and a blurred image and the respective regions are processed with an unsharp masking, a Laplacian filter or other suitable means using different coefficients such that graininess is suppressed in the flat region whereas sharpness is enhanced in the contour region, thereby achieving graininess suppression without producing blurry edges. A problem with this technique is that discontinuity will occur at the boundary between the contour and grainy regions because the recognition and separation of these regions are performed indiscriminately with reference to a single threshold signal level.
It should also be mentioned that in the prior art which employs unsharp masking or a Laplacian filter for edge or sharpness enhancement, fringe (over shoot) such as Mach bands are most likely to occur along the contour or edges of the image, giving the visual impression of artificiality.
The present invention has been accomplished under these circumstances and it relates to an improvement of the technology of restoring images such as those in photographs, printed documents, or on television""s screens, in digital still photographs and from various kinds of copiers so as to restore the camera-induced blur, the noise and the deterioration in sharpness which is inherent in the original picture as exemplified by graininess and blur in photographic materials or the noise and the deterioration in sharpness which has been added when digitizing the original picture with an image input device. As already mentioned, the prior art technology suffers three major problems: when a smoothing technique is applied to suppress graininess, the grain becomes blurred and mottling produces a visually unpleasing impression; low-contrast picture signals are mistaken for graininess and suppressed or eliminated; and the boundary between a region from which graininess has been removed and a region where sharpness enhancement has been done becomes discontinuous to produce unnatural artifacts in the output image.
An object of the present invention is to provide a method for processing digital images to suppress noise and enhance sharpness, by which graininess can be suppressed and image sharpness enhanced without causing any of the problems associated with the prior art.
Another object of the present invention is to provide an image processing apparatus for implementing the above method.
In order to achieve the first object, the present invention provides a method for processing a digital image for noise suppression and sharpness enhancement, comprising the steps of:
performing a sharpness enhancing process on original image data to create sharpness enhanced image data in which not only an image but also a grain or noise contained therein is sharpened;
performing a smoothing process on said original image data to create smoothed image data;
subtracting said smoothed image data from said sharpness enhanced image data to create first edge/grain composite image data comprising edges of a subject image and grains which are both enhanced in sharpness;
performing an edge detection of said original image data to determine an edge intensity data for discriminating an edge region of a subject and a grainy region;
using said edge intensity data to determine grainy fluctuation compressing coefficient data for compressing amplitude of an grainy fluctuation component in the grainy region;
multiplying said first edge/grain composite image data by said grainy fluctuation compressing coefficient data to compress only the grainy fluctuation component in the grainy region selectively, and to thereby create second edge/grain composite image data in which the grains in the grainy region are suppressed while retaining the sharpness of the edges in the edge region of the image; and
adding said second edge/grain composite image data in which the grains are suppressed while enhancing the sharpness to said smoothed image data to thereby create an processed image.
In order to achieve the second object, the present invention also provides an apparatus for processing a digital image for noise suppression and sharpness enhancement, comprising:
a sharpness enhancing unit for performing a sharpness enhancing process on original image data to create sharpness enhanced image data in which not only an image but also a grain or noise contained therein is sharpened;
a smoothing unit for performing a smoothing process on said original image data to create smoothed image data;
an edge/grain composite component extracting unit for subtracting said smoothed image data from said sharpness enhanced image data to create first edge/grain composite image data comprising edges of a subject image and grains which are both enhanced in sharpness;
an edge detecting unit for performing an edge detection of said original image data to determine an edge intensity data for discriminating an edge region of a subject and a grainy region;
a grain compressing coefficient calculating unit for determining grainy fluctuation compressing coefficient data for compressing amplitude of an grainy fluctuation component in the grainy region based on the edge intensity data;
a grainy component compressing unit for multiplying said first edge/grain composite image data by said grainy fluctuation compressing coefficient data to compress only the grainy fluctuation component in the grainy region selectively, and to thereby create second edge/grain composite image data in which the grains in the grainy region is suppressed while retaining the sharpness of the edges in the edge region of the image; and
an output image calculating unit for adding said second edge/grain composite image data in which the grains are suppressed while enhancing the sharpness to said smoothed image data to thereby create an processed image.
Preferably, said grainy fluctuation compressing coefficient data has a value equal or close to 1.0 in said edge region of said subject and is gradually decreased from said edge region toward said grainy region, where said compressing coefficient data takes a specified value ranging between 0.0-1.0.
Preferably, said grainy fluctuation compressing coefficient data is represented by equation (1):
CG=(x,y)=(1xe2x88x92kG)E0(x,y)+kGxe2x80x83xe2x80x83(1)
where CG (x,y) denotes the grainy fluctuation compressing coefficient data, E0 denotes said edge intensity data (normalized E0 (0xe2x89xa6E0xe2x89xa61)), and kG denotes a grain compressing constant for adjusting the degree of compression of the grainy component in the grainy region and is a value ranging between 0.0-1.0.
Preferably, said edge detection is performed by one of a local variance method, a spatial first differential method and a spatial second differential method.
Preferably, said sharpness enhancing process is one of Gaussian unsharp masking, differentiation filtering and spatial frequency filtering.
Preferably, said smoothing process is one of processes directed to a real space domain and a spatial frequency domain.
Said edge detection, said sharpness enhancing process and said smoothing process are more preferably performed by a local variance method, Gaussian unsharp masking and Gaussian blurry masking, respectively. The processing is not limited to the Gaussian type, and any other type of processing may of course be used.
Said sharpness enhancement is preferably applied in a sufficiently intense manner, even though the image has a rather marked graininess without suppressing the graininess.
Preferably, the grainy fluctuation compressing coefficient data CG (x,y) is used to perform a weighting operation in the edge region of said subject, and the grainy region does not abruptly separate the edge region from the grainy region on the boundary thereof with two values of on/off, but gradually changes relative proportions of the two regions.
Preferably, said method for suppressing the graininess comprises performing the sharpness enhancing process on the original image data to make a grainy pattern spatially finer and to reduce the amplitude of the grainy fluctuation component. The spatial fining and amplitude reduction which correspond to fining of silver halide gains as photo-sensors in a silver halide photographic material, can provide a visually finer and pleasing granularity.
The image processing method and apparatus of the invention have the following effects:
1) An original image of which the sharpness was deteriorated due to a camera lens, a photographic film or optics including a scanner for digitizing films is subjected to the sharpness enhancing process to sharpen the subject image selectively.
2) A blurred and expanded grainy pattern is sharpened by the sharpness enhancing process. Further, a grainy pattern which is spatially finer irrespective of signal variation and which is pleasing to the eye can be obtained by reduction of the amplitude of the grainy fluctuation signals. This corresponds to fining of silver halide grains in silver halide color films.